Making Sense of Meta-Analysis: A Critique of "Effectiveness of Long-Term Psychodynamic Psychotherapy"

Evidence-based practice depends in part on knowledge derived from relevant research. For any given topic, there are likely to be many, potentially relevant studies; a careful appraisal and synthesis of the results of these studies is needed to understand the state of the empirical evidence. Meta-analysis is widely used to combine results of quantitative studies; yet this method is unfamiliar to many people and, as a result, meta-analyses are often uncritically accepted. In this article, we argue that meta-analysis is only one component of a good research synthesis. We critique a recent metaanalysis on the effectiveness of long-term psychodynamic psychotherapy, showing that this metaanalysis failed to meet current standards for the conduct and reporting of systematic research reviews and meta-analyses. We demonstrate the use of AMSTAR, a straightforward tool for assessing the quality of systematic reviews and meta-analyses. Making sense of meta-analysis 3 Evidence-based practice depends in no small measure on the identification, critical appraisal, and accurate synthesis of relevant research results. Methods of research synthesis have come a long way since Glass (1976) coined the term “meta-analysis” to describe the statistical combination of data from multiple studies. Meta-analysis is now widely used in the social, behavioral, and health sciences; yet, many people are unfamiliar with this approach and, as a result, meta-analyses are often uncritically accepted. In this article, we critique a recent meta-analysis on the effectiveness of long-term psychodynamic psychotherapy (LTPP) by Leichsenring and Rabung (2008). This meta-analysis appeared in JAMA, where Glass himself pronounced that it “was carefully performed” (2008, p. 1589) and others disagreed (Beck & Bhar, 2009; Kriston, Holzel, & Harter, 2009; Glass, 2008; Roepke, 2009; Thombs, Bassel, & Jewett, 2009). We argue that meta-analysis is only one component of a systematic approach to reviewing research, and the LTPP meta-analysis failed to meet current standards for the conduct and reporting of systematic reviews and meta-analyses. The LTPP meta-analysis is controversial in terms of its content as well as its methods. For many years now, the treatment and research worlds have witnessed the ascendancy of short-term cognitive and behavioral therapies. At face value, the article by Leichsenring and Rabung calls into question the move toward short-term treatment of a range of mental health issues. As evidence-based practice has developed over the years, so too have the tools needed to quickly and competently evaluate the presentation of findings from different types of studies, including random controlled trials, studies of diagnostic accuracy, observational studies, systematic reviews, and meta-analyses (see http://www.equatorMaking sense of meta-analysis 4 network.org). There are now also user-friendly tools to evaluate the quality of completed studies. We demonstrate the application of just such a tool (AMSTAR; Shea et al., 2007) for assessing the quality of systematic research reviews.